Medical training systems and methods

ABSTRACT

Simulation systems and methods may enable virtual imaging. A data processing unit may receive data from a calibration unit indicating a position and/or orientation of a position and orientation sensor relative to a physical model. The data processing unit may also receive data from the position and orientation sensor indicating a position and/or orientation of the physical model. The data processing unit may generate a virtual image using the data from the position and orientation sensor and the data from the calibration unit. The data processing unit may render the virtual image to a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/589,714, filed Jan. 23, 2012, which is incorporatedby reference in its entirety.

BACKGROUND

1. Field

The present invention relates to computerized medical simulation ingeneral, and more specifically to virtual reality and/or augmentedreality simulation devices and systems for medical training purposes.

2. Technical Background

Traditional medical procedures requiring invasive surgery are frequentlybeing replaced by less invasive image-guided techniques taking advantageof modern imaging technologies, such as endoscopy, for both diagnosticand therapeutic purposes. These new techniques may require dedicatedtraining for physicians and surgeons to master the indirect hand-eyecoordination required by the imaging system as well as the manipulationof the imaging tools in addition to the conventional medical instrumentsand procedures. Computerized medical procedure training simulators mayenable the physicians and trainees to develop and improve their practicein a virtual reality environment before actually practicing in theoperation room.

Advanced medical procedure simulators may be based on a virtual reality(“VR”) and/or a mixed or augmented reality (“AR”) simulation apparatusby which the physician can experiment a medical procedure scenario. TheVR/AR system may compute and display a visual VR/AR model of anatomicalstructures in accordance with physician gestures and actions to providevarious feedback, such as visual feedback. In a VR system, an entireimage may be simulated for display to a user, and in an AR system, asimulated image may be overlaid or otherwise incorporated with an actualimage for display to a user. Various patient models with differentpathologies can be selected. Therefore, natural variations asencountered over the years by practicing surgeons can be simulated for auser over a compressed period of time for training purposes. The medicalsimulation procedure can be recorded and rehearsed for evaluationpurpose. The VR/AR simulation system can also compute and providevarious metrics and statistics.

Some VR/AR simulation systems such as the one described in patentapplication US2010/0086905 include a human anatomy model of a joint oforgan in real size and a simulated medical instrument that imitates thereal medical procedure instrument. The model is further adapted withsensors and mobile members for guiding, tracking and controlling themedical instrument operation within the anatomy model. The simulatedmedical instrument is usually adapted from dedicated VR or AR hardwareto provide haptic feedback, such as force feedback, to the physician toreproduce the instrument touch and feel sensing when touching the model.

Legacy medical VR/AR simulators often require dedicated haptic hardwaresuch as, for instance, the two haptic devices arranged with a mobilemember in the arthroscopy VR simulator described in U.S. patentapplication 2010/0086905. Those VR/AR simulators can require specificsetup and calibration prior to be operated; they can be expensive; theymay suffer a number of mechanical constraints, such as friction, and themaximal amount of forces and torques displayed may be insufficient toprovide a realistic simulation. The degree of possible motion of thetools due to mechanical configuration of the device can be limited.Moreover, the setup can be unstable over time, thus requiring regularmaintenance to avoid inducing training errors. Furthermore, for amultipurpose training room, many simulators may be required, asdifferent anatomical features may require different simulators. Forinstance, a knee arthroscopy simulator and a pelvic simulator mayrequire two different simulators to be set up and calibrated.Furthermore, in order to align a VR/AR computerized simulation visualfeedback to an actual physical gesture and action of the simulationtools, the VR/AR system may require a calibration setup phase toposition the VR/AR model relative to real world position and orientationof each simulated instrument before the simulator becomes operational.Wrong calibration should be carefully avoided as it can be a source ofsignificant training errors, potentially harmful to patients in laterreal operations. As taught for instance by U.S. patent application2010/0248200, the haptic device is therefore often manually calibratedby asking one or more experienced physician or medical residents totouch all virtual tissues using a virtual medical procedural tool, whichmay for example be a virtual blade, controlled by the haptic device.

The training tools described above can create significant budget andtime overhead for the multipurpose training room operation, as multiplededicated hardware devices may need to be installed in the room andskilled personnel may be required to operate the devices. Switching fromone medical training procedure to another may be consequently cumbersomeand time-consuming.

SUMMARY

Embodiments of flexible medical simulation apparatus, methods, andsystems described herein may automatically adapt to various medicalprocedure training scenarios without requiring specialized hardwaresetup and calibration in the training room. A multipurpose medicalprocedure VR/AR simulator may be set up using an anatomy modelcomprising at least one position and orientation sensor and acalibration unit associated with such sensor. The calibration unit maystore and transmit pre-computed calibration data associated with thesensor to the VR/AR simulator data processing unit. The data processingunit may compute and display an accurate VR/AR model in accordance withthe anatomy model sensor position and orientation on the one hand andthe VR/AR training scenario on the other hand. Accordingly, variousVR/AR training scenarios corresponding to different patient pathologiescan be simulated in a highly realistic way without requiring costlydedicated hardware integration, calibration, setup, and maintenance.

A multipurpose medical procedure VR/AR simulator may be setup with aninterchangeable anatomy model comprising at least one position andorientation sensor and a calibration unit associated with such sensorand such model. The anatomy model may be plugged into the VR/ARsimulator via a fast mount electrical and mechanical connection, forexample. The calibration unit may store and transmit an anatomy modelidentification as well as pre-computed calibration data associated withthe anatomy model sensor to the VR/AR simulator data processing unit.The data processing unit may compute and display an accurate VR/AR modelin accordance with the anatomy model type, the sensor position andorientation, and the VR/AR training scenario. Accordingly, various VR/ARtraining scenarios corresponding to different anatomy models anddifferent patient pathologies can be simulated in a highly realistic waywithout requiring a different medical procedure VR/AR simulator cart foreach scenario or for some subsets of scenarios.

A medical procedure VR/AR simulator equipped with standard operatingroom tools comprising at least one position and orientation sensor and acalibration unit associated with such sensor and tool may facilitate thetraining of physicians. Each medical tool may have a sensor that may beplugged onto the VR/AR simulator via a standard, fast mount electricaland mechanical connection, for example. The calibration unit may storeand transmit the tool identification as well as the pre-computedcalibration data associated with the tool sensor to the VR/AR simulatordata processing unit. The data processing unit may compute and displayan accurate VR/AR model in accordance with the anatomy model type, eachanatomy model sensor position and orientation, each tool type, each toolsensor position and orientation, and/or the VR/AR training scenario.Accordingly, various VR/AR training scenarios corresponding to differentanatomy models, different patient pathologies, and/or different medicaltools can be simulated in a highly realistic way with real tools.Moreover, different medical tools can be adapted to the VR/AR simulatorat any time without requiring costly dedicated hardware integration,calibration, setup, and maintenance by highly skilled personnel.

The daily operation and maintenance of a VR/AR simulation training roommay be facilitated by applying relative sensor position and orientationmeasurement to avoid cumbersome VR/AR simulator sensors calibration. Tothis end, more sensors than are theoretically needed for a fulldetermination of the VR/AR model orientation and position may beemployed in order to combine and compute the relative positions andorientations of the sensors in addition to and/or instead of theirabsolute position and orientation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a medical procedure VR/AR simulator cart hosting ananatomy model according to an embodiment of the invention.

FIG. 1B illustrates a medical procedure VR/AR simulator cart hosting ananatomy model according to an embodiment of the invention.

FIG. 2A illustrates a partial cutaway view of a medical simulation carthosting an anatomy model according to an embodiment of the invention.

FIG. 2B illustrates a partial cutaway view of a medical simulation carthosting an anatomy model according to an embodiment of the invention.

FIG. 3A illustrates a partial cutaway view of a medical simulation carthosting an anatomy model and at least one medical tool according to anembodiment of the invention.

FIG. 3B illustrates a partial cutaway view of a medical simulation carthosting an anatomy model and at least one medical tool according to anembodiment of the invention.

FIG. 4 illustrates a set of medical tools adapted to comprise at leastone sensor and a calibration unit according to an embodiment of theinvention.

FIG. 5 illustrates a number of different VR/AR medical training usecases according to an embodiment of the invention.

FIG. 6 illustrates a flow chart of a self-calibration method accordingto an embodiment of the invention.

DETAILED DESCRIPTION

FIGS. 1A-1B represent a medical procedure VR/AR simulator cart 10comprising a data processing unit 100, a display screen 110, and a plug120 adapted to receive a human anatomy model 130, according to anembodiment of the invention. For the purpose of illustration, in FIGS.1A-1B, a human anatomy model 130 of a knee joint is shown, but othermodels can be used as well. The human anatomy model 130 may be made offlexible plastic or any other suitable material. The knee model of FIGS.1A-1B can be manipulated with the handle 135 to reproduce model motionand deformation. For example, the knee model 130 can be manipulated tosimulate various leg flexion angles and/or various valgus-varus angles.

The anatomy model 130 may be fastened to the cart 10 in such a way thatit may be easily plugged in and out, for instance by clipping it ontothe cart, while reproducing a real organ position in the operating room.In the embodiment of FIGS. 1A-1B, the knee model 130 is fastened to avertical mount 115 that is itself plugged onto the cart plug 120 so thatit may simulate a position of a real organ in the operating room.

In the arthroscopy application field, in addition to the knee, otherjoint models may be used and interchanged with the human anatomy model130, such as a shoulder model, a hip model, a wrist model, or an elbowmodel. Furthermore, in order to support other medical procedures thanarthroscopy, other organ models may be interchanged with the anatomymodel 130, such as a bladder, a womb, an upper torso, a lower torso, ora pelvic model. Other examples of models can be found in the cataloguesof specialized anatomic model suppliers such as Limbs&Things, Bristol,UK. Some models may be representative of the human anatomy. Other modelsmay be representative of an animal anatomy, e.g. for veterinary trainingpurpose. Some models can be attached directly to the cart plug 120without a vertical mount 115, for instance a lower torso or pelvicmodel. Other models may be fastened to a vertical mount 115 that isattached to the cart plug 120.

The data processing unit 100 may comprise a central processing unit(“CPU”), memory, controlling module, and/or communication module, forexample. Other embodiments may include data processing units 100 withother configurations and combinations of hardware and software elements.A distributed data processing unit may be used. Some or all of the dataprocessing unit 100 components may be used to compute and display onto adisplay screen 110 a VR/AR simulation model that may correspond to achosen medical procedure training scenario. Multiple display screens mayalso be used. The display screen 110 may comprise a touch interface toprovide an interface for a physician during a simulation exercise. FIG.1A illustrates the display screen 110 in a lowered position, which maybe suitable for storage or transport, for example. FIG. 1B illustratesthe display screen 110 in a raised position, which may be suitable foroperation during a medical training scenario, for example. In someembodiments, the display screen 110 may be configured such that it canbe positioned in one or more positions or viewing angles in addition tothose shown in FIGS. 1A-1B. In other embodiments (not illustrated) thesimulator cart may further comprise a camera.

FIGS. 2A-2B depict a partial cutaway view of a medical simulation cart10 hosting an anatomy model 130 according to an embodiment of theinvention. Like FIGS. 1A-1B, FIG. 2A shows the medical simulation cart10 with the display screen 110 in a lowered position, and FIG. 2B showsthe medical simulation cart with the display screen 110 in a raisedposition. In FIGS. 2A-2B, the human anatomy model 130 comprises at leastone position and orientation sensor 200 and at least one calibrationunit 210 associated with the sensor 200. In FIGS. 2A-2B where a kneejoint model 130 is illustrated, two position and orientation sensors200, 201 are integrated in order to measure the position and orientationof the two main bones forming the knee joint, that is the tibia and thefemur. For example, six degree of freedom (“6DOF”) miniaturized magnetictracking sensors 200, 201 may be mounted into the anatomy model 130, anda magnetic sensor transmitter unit 220 may be integrated into the mount115 or directly into the anatomy model 130 (not illustrated). Themagnetic sensor transmitter unit 220 may generate a magnetic fieldnecessary for proper operation of the sensors 200, 201. The sensors 200,201 may be substantially identical sensors so that interference fromother fields or field distortion may be reduced or eliminated throughcommon mode rejection. Substantially identical sensors may be sensorsthat are the same sensor model or have the same part number, forexample. Each sensor 200, 201 may be connected to a tracking unit 230through a dedicated communication link. In some embodiments the mount115 may be fastened to the cart plug 120 in such a way that it is easilyelectrically connected in and disconnected out, for instance via a USBconnection or other standard connection. Other embodiments are alsopossible, e.g. the tracking unit 230 can be integrated within theanatomic model 130 and/or the mount 115.

The tracking unit 230 may receive the sensor information and transmit itto the data processing unit 100 through a connection 235 such as a USBlink, other standard connection, or other communication link. The dataprocessing unit 100 may use the tracking unit 230 inputs to calculatethe virtual anatomy model 130 position and orientation in accordancewith the sensor 200, 201 measurement. The data processing unit 100 mayuse the calculated model 130 position and orientation to generate avisual model and display the visual model onto the display screen 110.

As known to those skilled in the art, the VR/AR model may be alignedwith the anatomy model based on assumptions on the anatomy model actualposition and orientation e.g. relative to the transmitter, and a sensormay be attached to the moving part of the anatomy model only, so that anabsolute position and orientation of the VR/AR model can be derived bythe VR/AR simulator. This solution is subject to drifts and instabilitydue to, for instance, magnetic field distortion induced by a metalobject or other electrical medical appliances in the close area of theVR/AR simulator. Thus, highly accurate tracking, which may be useful forsome applications such as knee arthroscopy simulation, may requireregular, cumbersome calibration of the sensors in the daily trainingroom practice. In embodiments described herein, a relative, over-definedposition and orientation of the model may be determined by calculatingthe location of at least one reference sensor 200 attached to the fixedpart of anatomy model 130. When the anatomy model 130 also includesmoving parts, the reference sensor 200 may complement the sensors thatare used to track each moving part of the model, for instance the kneesensor 201 attached to the moving part of anatomy model 130.

In FIGS. 2A-2B, the tracking unit 230 may detect the position of sensor201 to track the moving tibia part of the anatomy model 130 and theposition of sensor 200 to track the fixed femur part of the knee model130. Depending on the mechanical properties and ergonomic constraints ofthe anatomy model 130, various alternative mount positions may be chosenfor sensors 200 and 201. Consequently, each sensor 200, 201 may have adifferent position and orientation relative to the other fixed parts ofthe anatomy model 130. For instance sensor 201 may have been attached atvarious places along the tibia bone in the case of the knee joint ofFIGS. 2A-2B depending on how the model 130 has been manufactured, andsensor 201 may have a different position and orientation relative to thefemur bone in the knee mode 130 of FIGS. 2A-2B and the tracking unit230.

The resulting calibration data may therefore differ from one anatomymodel 130 to another. In the case of the knee joint depicted on FIGS.2A-2B. depending on how the anatomy model 130 is positioned whenattached to the mount 115, the initial VR/AR model display generated bythe data processing unit 100 may have to show a different leg flexionand varus-valgus angle matching the actual anatomy model 130 positionand orientation. Prior accurate knowledge of the actual sensor 200, 201positioning with regards to the anatomy model 130 bones, that is thesensor calibration, may be used to accurately determine the position andorientation of the VR/AR model. In the example of the knee model 130,accuracy in the order of magnitude of less than 1 mm in the 6DOFpositioning may be observed, instead of up to 1 cm erroneous measurementexperienced in other passive feedback solutions.

In accordance with the embodiments described herein, at least one sensor200 may be integrated into the fixed part of the anatomy model 130. Inaddition, for models 130 including at least one moving part, at leastone sensor 201 may be integrated within movable parts of the model 130.For example, each rigid part of the anatomy model 130 that movesrelatively to another rigid part in that particular model 130, such asthe tibia bone relative to the fixed femur bone in the knee arthroscopycase, may have its own sensor 201. Within the anatomy model 130, thetissues and the parts of the organs that are subject to differentpathology case modeling, such as various shapes of ligaments andmeniscus, may be simulated by various different VR/AR modelscorresponding to different types of patients, for instance an adult or achild, and/or different pathologies training cases. In addition, theposition and orientation of the rigid and non-rigid structures withoutsensors 201 can also be interpolated based on information from otherknown structures, therefore the choice and placement of sensors 201 canbe chosen according to accuracy and cost requirements of the givenapplication.

In some embodiments, the sensor calibration data may be pre-computed atthe time of sensor 200, 201 integration into the anatomy model 130 inaccordance with the target VR/AR simulation model and stored into acalibration unit 210 associated with the anatomy model 130. In the caseof the knee joint depicted in FIGS. 2A-2B, sensor 200 data maycorrespond to the fixed femur bone position and orientation as mountedonto the vertical mount 115 and sensor 201 data may correspond to theflexible tibia bone whose position and orientation is variable relativeto the fixed femur. The calibration unit 210 may comprise storage andcommunication elements to store the sensor 200, 201 calibration dataand/or transmit the sensor 200, 201 calibration data to the dataprocessing unit 100.

For example, the calibration unit 210 may be implemented as a USB dongleconfigured to be connected to the sensor 201 via an electricalconnection 215 such as a wire and plugged into data processing unit 100.The USB dongle calibration unit 210 may also be connected to the dataprocessing unit 100 only or to the sensor 201 only. Other embodimentsare possible as well, for instance the calibration unit 210 may bedirectly integrated into the anatomy model 130 and connected to the dataprocessing unit 100 through the electrical fastening of the mount 115onto the cart plug 120, or through an electrical fastening of the cartplug 120 itself, or through a wireless connection between thecalibration unit 210 and the data processing unit 100.

The data processing unit 100 may compute the VR/AR model position andorientation by combining the absolute sensor measurement received fromtracking unit 230 with the pre-computed calibration data from thecalibration unit 210 matching the simulated VR/AR model. While in FIGS.2A-2B the calibration unit 230 is represented as a separate unit fromthe model 130, in alternate embodiments it may be part of the model 130.In some embodiments in which the calibration unit 230 is integrated withthe model 130, calibration data from the calibration unit 230 and fromthe sensor(s) 200, 201 may be transmitted over the same medium (e.g.,the same physical wire(s) or same wireless frequency(ies)) and/or may betransmitted by the same transmitter.

In some embodiments, fur example when the calibration unit 210 does notcommunicate directly with the data processing unit, the tracking unit230 may store and transmit the anatomy model 130 identification from thecalibration unit 210 to the data processing unit 100 so that the dataprocessing unit 100 can automatically determine which model has beenmounted onto the VR/AR simulation cart and propose a choice of matchingVR/AR simulation training scenarios accordingly. The identification maycomprise a model type, version, and serial number. For example, theproposed scenario may be displayed on the display 110, and a user mayselect the proposed scenario or a different scenario via the touchscreenor other interface.

It may be possible to simulate a number of different procedures over aphysical model in a highly realistic way by further appending standardmedical tools onto the VR/AR simulator cart of FIGS. 2A-2B, as will nowbe described in more detail with reference to FIGS. 3A-3B.

FIGS. 3A-3B show a medical tool 300 appended to the medical simulationcart of FIGS. 2A-2B. Like FIGS. 1A-1B and 2A-2B, FIG. 3A shows themedical simulation cart 10 with the display screen 110 in a loweredposition, and FIG, 3B shows the medical simulation cart with the displayscreen 110 in a raised position. In many medical procedure simulationsat least one imaging tool may be needed, for instance to simulate adiagnosis operation by allowing exploration of the organ. Examples ofimaging tools may include endoscopes that are inserted directly into theorgan by a natural orifice or through a small incision or imaging probessuch as ultrasound probes that can also be used externally. For thepurpose of illustration, in the case of a knee joint arthroscopysimulation, the medical tool 300 of FIGS. 3A-3B may be an arthroscopethat can be inserted into the joint anatomy model 130 through a portal310. The portal 310 may be chosen among known medical portals such as,in the case of the knee, the antero-medial portal, the antero-lateralportal, the dorso-medial portal, the dorso-lateral portal, thesupramedial portal, or the dorso-supralateral portal. In someembodiments, the imaging tool 300 may be a standard operation room toolsuitable for various medical procedures, for instance an arthroscopeindifferently suitable for knee, shoulder, or elbow endostopy, that maybe adapted to comprise at least one position and orientation sensor 320and one calibration unit 330 associated with sensor 320. Depending onthe anatomic model used, the portal or portals 310 might not be needed,for instance when a natural orifice can be used, or might be alreadyincorporated in the models, or might be created by the operator at anyconvenient position.

The tool sensor 320 may be connected to the tracking unit 230 through adedicated link 325. The tracking unit 230 may receive the sensorinformation and transmit it to the data processing unit 100 through astandard connection 235 such as a USB link or via some othercommunication channel. The data processing unit 100 may use the trackingunit 230 inputs to calculate the virtual anatomy model position andorientation in accordance with the model sensor 200, 201 measurement andthe tool sensor 320 measurement respectively. The data processing unit100 may use the calculated model 130 position and orientation togenerate a visual model and display the visual model onto the displayscreen 110.

Physical parts of the anatomy model 130 that can be moved independentlyalong at least one degree of freedom may require accurate tracking toavoid real world collisions of the tool with the anatomy model duringthe training manipulation, according to the various possible VR/ARmodels corresponding to different patients and different pathologies.Therefore, a sensor 201 may be integrated into each part of the anatomymodel 130 that can be moved independently. It may be useful to provideas accurate a measurement as possible of the relative positions andorientations of the sensors 201 with regard to the physical model parts.This accuracy may be achieved by pre-calibrating each sensor 201.

As known to one skilled in the art, initial alignment of the VR/AR modelwith the position and orientation of the tool may also requirecalibration. For instance, in the case of the knee joint and arthroscopedepicted in FIGS. 3A-3B, depending on how the anatomy model ispositioned in the beginning, and how and where the arthroscope isinserted into the anatomy model 130 through portal 310, the initialVR/AR model display may show a different leg flexion and varus-valgusangle matching the actual anatomy model position on the one hand and theendoscope camera simulated field of view on the other hand. In someembodiments, calibration data may be pre-computed at a time when thesensor 320 is initially adapted onto the tool 300, and stored into acalibration unit 330 associated with the tool sensor 320. The standardoperation room tool 300 may be one that is useable in real medicalprocedures, and may therefore not be initially constructed with anorientation sensor 320 and calibration unit 330. The orientation sensor320 and/or calibration unit 330 may therefore be incorporated into thetool 300 after tool 300 manufacture, and the calibration may beperformed at this time. Other tools 300 may be dedicated training toolswhich may be constructed with a built-in orientation sensor 320 and/orcalibration unit 330, and calibration may be performed at constructiontime. For example, some expensive tools such as cameras may be simulatedby non-functional training tools. In the case of the arthroscope 300depicted in FIGS. 3A-3B, the sensor 320 is shown mounted at thehand-side end of the arthroscope 300, but depending on the mechanicalproperties and ergonomic constraints of the tool, various alternativemount positions may be chosen, for instance at the tip of the tool. Theresulting calibration data, which may define the proximity of the sensorto the tool components such as the tip or working element, may differaccordingly.

The calibration unit 330 may comprise memory and communication elements335 which may be configured to store the calibration data and/ortransmit the calibration data to the data processing unit 100.

The calibration unit 330 may also store the tool 300 identificationand/or transmit the tool 300 identification to the data processing unit100 so that the data processing unit 100 can automatically determine andsimulate an appropriate tool. For example, the tool identification maycomprise a tool 300 type and tool 300 manufacturer information.Additional tool 300 identification information may be supplied as well,such as a tool 300 version or model number. In addition, the calibrationunit 330 may also store parameters characterizing the behavior of thetool 300 (e.g. cutting speed in case of an arthroscopic shaver) andappearance of the tool 300, such as the geometric data, volumetric andsurface models, texture information, and any other information useful todescribe the tool in the VR/AR simulation.

In a possible embodiment as shown by FIGS. 3A-3B, the calibration unit330 may be implemented as a USB dongle to be plugged into dataprocessing unit 100. While in FIGS. 3A-3B the calibration unit 330 isrepresented as a separate unit from the tool 300, in alternateembodiments it may be part of the tool 300. In some embodiments in whichthe calibration unit 330 is integrated with the tool 300, calibrationdata from the calibration unit 330 and from the sensor 320 may betransmitted over the same medium (e.g., the same physical wire(s) orsame wireless frequency(ies)) and/or may be transmitted by the sametransmitter.

At run-time in the training room, the data processing unit 100 maycompute the VR/AR model position and orientation by combining theabsolute sensor measurement received from tracking unit 230 with thepre-computed recorded calibration data from the calibration units 210,330 matching the actual simulated VR.AR model.

The combination of sensor and calibration unit disclosed above for theanatomy model and the imaging tool respectively can be generalized toany type of medical tool suitable for medical procedure simulationtraining. FIG. 4 illustrates sample tools that may be adapted tocomprise a position and orientation sensor 320 and a calibration unit330 with storage, processing, and communication elements to handle theposition and orientation calibration data of said sensor 320. Thesesample tools include a hook to push and pull tissue 301, a shaver 302,and a grasp handle 303. Other tools, not illustrated herein, can beadapted as well, such as a palpation hook, various types of punches(straight, slightly bent up, bent up, bent left, bent right and 90°), agrasp, a shaver hand-piece with flow control, a foot pedal entity, apump, an ECG or EEG monitoring probe, etc.

Some medical tools may be equipped with buttons or various mechanicalinteraction elements which may be manipulated by the end user, forinstance a foot pedal or an imaging tool zoom. In some embodiments, forproper VR/AR modeling and rendering, the tool 300 may report thereal-time user interaction information to the data processing unit 100through the calibration unit 330, in addition to the tool identifier andcalibration data.

Other embodiments of the described VR/AR simulation cart, tools andsystem are also possible. Various sensor technologies may be used, suchas the magnetic 6DOF tracking sensors suitable for medical use assupplied for instance by the Ascension manufacturer, or optical trackingsensors. The sensors 200, 201, 320 may be directly connected to the dataprocessing unit 100 without a sensor transmitter unit 220 or trackingunit 230. When magnetic tracking sensors are used, the transmitter unit220 may be mounted directly into the anatomy model 130, the mount 115,or the cart. When optical tracking sensors are used, a camera ratherthan a transmitter unit may be used and may be mounted in differentplaces. For example, the camera may be placed far enough above the VR/ARsimulator cart to allow the optical tracking sensors to substantiallyremain in the camera's field of view during the training manipulation.Some or all connections described herein may be wired or wireless, forinstance by means of power efficient wireless standards such as Zigbee.In particular, wireless communication may be used for adapting medicaltools already using wireless connectivity in the operation room. In manycases it may be desirable to use sensors measuring all six degrees offreedom of position and orientation, in certain cases it may be possibleto use simpler sensors measuring less than six degrees of freedom, forinstance in a case where the anatomy model is constrained in a certainposition and/or orientation dimension it may be possible to measure asingle degree of freedom such as a rotation angle.

In one embodiment, one calibration unit may be associated with eachsensor. In other embodiments, one calibration unit may store andtransmit the calibration data for at least two sensors. In someembodiments, the sensors 200, 201, 320 and the calibration units 210,330 may be combined into a single unit that may be mounted onto thephysical anatomy model, the mount, or the tool. Although depicted asseparate in the above illustrations, any of the calibration unit 210,the sensor transmitter unit 220, the tracking unit 230, and the dataprocessing unit 100 may also be combined.

The VR/AR simulation apparatus, systems, and methods described hereinmay enable the setup and operation of a multi-purpose training room, aswill become more apparent from the description of VR/AR simulationinitialization methods. Various medical training scenarios can besimulated using the same multi-purpose VR/AR simulation cart bymounting, with reference to FIGS. 2A-2B, the anatomy model 130 and mount115 corresponding to the target medical training scenarios onto thesimulation cart plug 120, connecting the anatomy model sensors 200, 201to the tracking unit 230, and connecting the calibration unit 210 to thedata processing unit 100. Further medical training scenarios can besimulated using the same multi-purpose VR/AR simulation cart by furtherconnecting, with reference to FIGS. 3A-38 and FIG. 4, the tool sensor320 to the tracking unit 230 and the tool sensor calibration unit 330 tothe data processing 100 for each relevant medical tool 300, 301, 302,303, according to the target scenario. It therefore may be possible foran apprentice physician to train on different anatomy models, differentmedical tools and different medical procedures using the same VR/ARsimulation system cart, without requiring substantial hardware setup andcalibration process by skilled personnel for each different trainingscenario.

Moreover, when new anatomy models or medical tools become available tobe integrated into the training room, it may be possible to order anewly adapted model or medical tool from a training simulator providerand connect it to the VR/AR simulation cart without requiring on sitespecialized personnel for initial setup and training, in particular interms of calibration of the VR/AR modeling. In accordance with theembodiments described herein, the new model or tool may be adapted andconfigured off-site by pre-calibration and be delivered to the trainingroom operator with its “plug-and-train” pre-computed recordedcalibration data. In addition, if appearance and behavior of the tool isstored together with the calibration data, simulating new tools withoutchanging the data processing unit or simulation software may bepossible.

FIG. 5 depicts a number of different VR/AR medical training use casesaccording to an embodiment of the invention. As shown in FIG. 5, variouspathologies can be further simulated with different VR/AR simulationsmodels and various combinations of physical anatomy models and tools,for instance knee arthroscopy, shoulder arthroscopy, and pelvicendoscopy, with the same multipurpose VR/AR simulation cart 10. The dataprocessing unit 100 may identify an attached anatomy model 130 and tools300, 301, 302, 303 from their respective calibration unit identificationdata, select a choice of possible VR/AR training scenarios matching saidmodel and tools, and display the choice as a graphical menu onto thedisplay screen 110 for the user to select a scenario. For instance, inthe case of knee arthroscopy, various sizes of knee joints, variablearticular cartilage lesions, a partially or fully torn cruciateligament, various meniscus tears, hypertrophy of Hoffa's fat pad, etc.,may be simulated. In advanced scenarios, the user can virtually repairthe meniscus by cutting the torn meniscus parts, as well as anyfree-floating broken cartilage and bone parts met in variouspathologies, with the grasp tool, replicating the gesture of extractingfrom the anatomy model virtually cut meniscus pieces or free-floatingparts. In practice, tissue, cartilage, and bone pieces(floating/connected) may be obscuring a view, and this scenario may berendered for highly realistic simulation scenarios. To this end, thedata processing unit 100 may compute the appearance and the behavior ofthe ligaments in the simulated joint, for example in order to simulatedamaged or worn ligaments, in accordance with the measured movement ofthe knee model (flexion, varus-valgus), the arthroscope, and relatedcalibration data. In some embodiments, the data processing unit 100 mayhighlight a selected pathology scenario region of interest by renderingit onto the display screen 110 with a different color, texture, orhighlighted contour. While the above examples are described specificallyfor the knee arthroscopy case, they may also apply to other jointarthroscopy pathologies and medical simulation applications in general.The user may manually select a scenario using the touchscreen, a mouse,the keyboard, a switch or a combination of any those. Other controls mayalso be used, such as for instance vocal control.

A VR/AR simulation system setup and operation method according to anembodiment of the invention will now be described in more detail. First,the user or setup operator may mechanically plug one of theinterchangeable anatomy models 130 either directly or by its verticalmount 115 onto the medical simulation cart plug 120, connect the anatomymodel sensor 200, 201 to the tracking unit 230 and connect the anatomymodel calibration unit 210 to the data processing unit 100. The operatormay further connect each relevant medical simulation tool sensor 320 tothe tracking unit 230 and each tool calibration unit 330 to the dataprocessing unit 100. In other embodiments, the setup according to theuser selection may also be automatized.

FIG. 6 represents a flow chart of a self-calibration method according toan embodiment of the invention.

In step 1, the data processing unit 100 may acquire the identifier datafrom each connected calibration unit 210, 330 to identify the connectedanatomy model 130 and any connected medical simulation tool 300, 301,302, 303. In step 2, the data processing unit 100 may select and displayon the display screen 110 a choice of medical simulation scenariosmatching the identified anatomy model and medical simulation tools. Instep 3, the operator may select a medical simulation scenario and inputthe selection to the data processing unit 100 which may receive theselection. For example, a touch screen element of the display screen 110may be used for input. In step 4, the data processing unit 100 mayacquire the pre-computed calibration data from each connectedcalibration unit 210, 330 for each position and orientation sensor 200,201, 320. In some embodiments, step 4 may be done at the same time asstep 1. The data processing unit 100 may then be ready to compute andrender the actual VR/AR simulation images. In step 5, the dataprocessing unit 100 may begin computing and rendering the VR/ARsimulation images by acquiring the orientation and position measurementdata for each sensor connected to the tracking unit 230 in real time. Instep 6, the data processing unit 100 may combine the orientation andposition measurement data with the pre-computed calibration data foreach sensor to determine the actual VR/AR simulation model position andorientation. For example, the pre-computed calibration data for eachsensor may be a six degrees of freedom position and orientation vectoroffset measuring the sensor position and orientation relative to a fixedreference predefined on the physical anatomy model or medical tool, asrelevant. In some embodiments additional tool interaction data may beinput in real time by the calibration unit 320 to the data processingunit 100 in order to compute and render a hilly interactive VR/ARsimulation scenario.

When all sensors are calibrated and can be tracked in real time, theVR/AR simulation that is rendered to the display 110 may substantiallycorrespond to the real positions of the various models and tools beingtracked. For example, if a user is manipulating a tool 300 and touches abone within the model 130, the display 110 may show the tool 300touching the bone at substantially the same instant and in substantiallythe same position of the actual contact. As noted above, accuracies onthe order of millimeters may be possible.

In some embodiments, the data processing unit 100 may estimate the VR/ARmodel position and orientation by weighting over-defined model sensors'position and orientation measurements according to sensor measurementreliability. In some embodiments, a same weight may be given to thedifferent sensors, so that the VR/AR model position and orientation maybe measured as an average of the different estimates. A reduced weightmay be allocated to a less reliable sensor, for instance when a magnetictracking sensor may be more exposed to the magnetic field distortion dueto its positioning within the VR/AR simulator anatomy model. A zeroweight may be allocated in the extreme case of a dysfunctional sensor.

In some embodiments, beyond live rendering of the VR/AR simulationscenario, the data processing unit 100 may apply various additionalprocessing, such as, but not limited to, recording and playing back thesimulation, evaluating and ranking the trainee performance and progress,compiling training statistics, raising alarms, and highlighting certainscenarios or scenes to draw special attention from the end user.

In order to provide highly realistic simulation scenarios, a highlydetailed rendering of the organ structure and texture corresponding todifferent pathologies, for instance flying pieces of tissues or brokenbones or ligaments, or bleeding, may be desirable.

In a case where multiple sensors are combined with complexinteractivity, for instance when simulating a use of a foot pedal tocontrol a shaver tool flow pressure into an organ, the data processingunit 100 may not be able to compute and render the highly realisticimages in real time due to the underlying VR/AR mathematical modelingcomplexity. To address this issue, a library of highly realistic imagesfor the most likely scenarios may be pre-computed offline and stored inthe data processing unit 100. The data processing unit 100 can theninterpolate and render the actual images in real time by interpolatingthem from the library images and the actual model, sensor, and tool datainputs.

Other advanced applications of the proposed VR/AR simulation system mayrequire the insertion of an external device into the anatomy model. Oneexample of such an application is the simulation of the insertion of anintrauterine device (“IUD”) into the womb through the cervix underultrasound control, or the simulation of the insertion of a coronarystent under fluoroscopy guidance. In some embodiments the externaldevice may be virtually simulated. In other embodiments, a real externaldevice may be used. In the latter case a position and orientation sensorand calibration unit can be mounted to the external device in a similarmanner to the tools and anatomical devices described above in order toaccurately track it in the VR/AR simulation scenario.

Advanced scenarios may require more direct interaction with the anatomymodel structures and tissue. For improved training realism, in someembodiments, it may be possible to combine the VR/AR simulation cart,tools, systems, and methods with haptic solutions such as force feedbackactuators. For example, haptic simulation may be provided to enable auser to “feel” a body part with the tool that is not physically presentwithin the model. The VR/AR display may show the haptically simulatedbody part at substantially the same time and in substantially the sameposition. Many example force feedback actuators may be known by thoseskilled in the art. In some embodiments, it may be possible to exploitand apply perceptual tricks to the VR/AR model simulation rendering sothat the end user experiences different virtual anatomic structureswithin the same physical anatomy model structures.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example and notlimitation. It will be apparent to persons skilled in the relevantart(s) that various changes in form and detail can be made thereinwithout departing from the spirit and scope. In fact, after reading theabove description, it will be apparent to one skilled in the relevantart(s) how to implement alternative embodiments. Thus, the presentembodiments should not be limited by any of the above-describedembodiments.

In addition, it should be understood that any figures which highlightthe functionality and advantages are presented for example purposesonly. The disclosed methodology and system are each sufficientlyflexible and configurable such that they may be utilized in ways otherthan that shown.

Although the term “at least one” may often be used in the specification,claims and drawings, the terms “a”, “an”, “the”, “said”, etc. alsosignify “at least one” or “the at least one” in the specification,claims and drawings.

Finally, it is the applicant's intent that only claims that include theexpress language “means for” or “step for” be interpreted under 35U.S.C. 112, paragraph 6. Claims that do not expressly include the phrase“means for” or “step for” are not to be interpreted under 35 U.S.C. 112,paragraph 6.

What is claimed is:
 1. A simulation system comprising: a data processingunit; a display in communication with the data processing unit; and aphysical model; wherein the physical model comprises: a position andorientation sensor configured and positioned to sense a position and/ororientation of the physical model; and a calibration unit configured tostore calibration data associated with the position and orientationsensor and the physical model; and the data processing unit isconfigured to: receive data from the position and orientation sensorindicating a position and/or orientation of the physical model; receivedata from the calibration unit indicating a position and/or orientationof the position and orientation sensor relative to the physical model;generate a virtual image using the data from the position andorientation sensor and the data from the calibration unit; and renderthe virtual image to the display.
 2. The simulation system of claim 1,wherein the physical model comprises a fixed portion, and wherein theposition and orientation sensor is attached to the fixed portion of thephysical model.
 3. The simulation system of claim 1, wherein the dataprocessing unit is configured to combine the virtual image with anactual image to form an augmented reality image.
 4. The simulationsystem of claim 1, wherein: the physical model comprises a plurality ofposition and orientation sensors; and the data processing unit isconfigured to: receive data from each of the plurality of position andorientation sensors indicating a position and/or orientation of each ofthe position and orientation sensors relative to a physical model; andgenerate the virtual image using the data from each of the plurality ofposition and orientation sensors and the data from the calibration unit.5. The simulation system of claim 4, wherein each of the plurality ofposition and orientation sensors is substantially identical.
 6. Thesimulation system of claim 4, wherein the physical model comprises amovable part. and at least one of the plurality of position andorientation sensors is disposed in the movable part.
 7. The simulationsystem of claim 4, wherein the data processing unit is furtherconfigured to weight the data from each of the plurality of position andorientation sensors in accordance with the data reliability.
 8. Thesimulation system of claim 1, further comprising the physical model. 9.The simulation system of claim 1, further comprising a mount configuredto accept the physical model.
 10. The simulation system of claim 1,further comprising a tracking unit in communication with the dataprocessing unit, the tracking unit configured to receive the data fromthe position and orientation sensor and transmit the data from theposition and orientation sensor to the data processing unit.
 11. Thesimulation system of claim 1, wherein the position and orientationsensor is a magnetic sensor.
 12. The simulation system of claim 11,further comprising a magnetic sensor transmitter unit configured togenerate a magnetic field detectable by the position and orientationsensor.
 13. The simulation system of claim 1, wherein the display is atouchscreen display.
 14. The simulation system of claim 1, wherein thephysical model comprises a model of an anatomical part.
 15. Thesimulation system of claim 1, further comprising a cart to which thedisplay is attached.
 16. The simulation system of claim 1, wherein thedata processing unit is configured to select a simulation scenario usingthe data from the position and orientation sensor and/or the data fromthe calibration unit.
 17. The simulation system of claim. 16, whereinthe data processing unit is configured to: display the simulationscenario on the display; and receive a selection of the simulationscenario.
 18. The simulation system of claim 16, wherein the dataprocessing unit is configured to generate a virtual image correspondingto the physical model using the data from the position and orientationsensor, the data from the calibration unit, and the simulation scenario.19. The simulation system of claim 1, further comprising an interfacefor communicating with a tool, wherein: the tool comprises: a toolposition and orientation sensor configured to sense a position and/ororientation of the tool; and a tool calibration unit configured to storecalibration data associated with the tool position and orientationsensor and the tool; and the data processing unit is configured to:receive data from the tool position and orientation sensor indicating aposition and/or orientation of the tool; receive data from the toolcalibration unit indicating a position and/or orientation of the toolposition and orientation sensor relative to the tool; generate a toolvirtual image corresponding to the tool using the data from the toolposition and orientation sensor and the data from the tool calibrationunit; and render the tool virtual image to the display.
 20. Thesimulation system of claim 19, wherein the position and orientationsensor and the tool position and orientation sensor are substantiallyidentical.
 21. The simulation system of claim 19, further comprising thetool.
 22. The simulation system of claim 19, wherein the tool comprisesa surgical tool.
 23. The simulation system of claim 19, furthercomprising a tracking unit in communication with the data processingunit, the tracking unit configured to receive the data from the toolposition and orientation sensor and transmit the data from the toolposition and orientation sensor to the data processing unit.
 24. Thesimulation system of claim 19, wherein the tool position and orientationsensor is a magnetic sensor.
 25. The simulation system of claim 19,wherein the data processing unit is configured to select a simulationscenario using the data from the tool position and orientation sensorand/or the data from the tool calibration unit.
 26. The simulationsystem of claim 25, wherein the data processing unit is configured to:display the simulation scenario on the display; and receive a selectionof the simulation scenario.
 27. The simulation system of claim 25,wherein the data processing unit is configured to generate a toolvirtual image corresponding to the tool using the data from the toolposition and orientation sensor, the data from the tool calibrationunit, and the simulation scenario.
 28. A simulation method comprising:receiving, with a data processing unit, data from a calibration unitindicating a position and/or orientation of a position and orientationsensor relative to a physical model; receiving, with the data processingunit, data from the position and orientation sensor indicating aposition and/or orientation of the physical model; generating, with thedata processing unit, a virtual image using the data from the positionand orientation sensor and the data from the calibration unit; andrendering, with the data processing unit, the virtual image to adisplay.
 29. The simulation method of claim
 28. further comprising:mechanically constraining, with a mount, a fixed portion of the physicalmodel to a fixed position relative to the mount,
 30. The simulationmethod of claim 28, further comprising: combining, with the dataprocessing unit, the virtual image with an actual image to form anaugmented reality image.
 31. The simulation method of claim 28, furthercomprising: receiving, with the data processing unit, data from each ofa plurality of position and orientation sensors indicating a positionand/or orientation of each of the position and orientation sensorsrelative to the physical model; and generating, with the data processingunit, the virtual image using the data from each of the plurality ofposition and orientation sensors and the data from the calibration unit.32. The simulation method of claim 31, wherein each of the plurality ofposition and orientation sensors is substantially identical.
 33. Thesimulation method of claim 31, wherein the data from each of theplurality of position and orientation sensors is weighted in accordancewith the data reliability.
 34. The simulation method of claim 28,further comprising: receiving, with a tracking unit in communicationwith the data processing unit, the data from the position andorientation sensor; and transmitting, with the tracking unit, the datafrom the position and orientation sensor to the data processing unit.35. The simulation method of claim 28, further comprising generating,with a magnetic sensor transmitter unit, a magnetic field detectable bythe position and orientation sensor.
 36. The simulation method of claim28, further comprising selecting, with the data processing unit, asimulation scenario using the data from the position and orientationsensor and/or the data from the calibration unit.
 37. The simulationmethod of claim 36, further comprising: displaying, with the dataprocessing unit, the simulation scenario on the display; and receiving,with the data processing unit, a selection of the simulation scenario.38. The simulation method of claim 36, further comprising generating,with the data processing unit, a virtual image corresponding to thephysical model using the data from the position and orientation sensor,the data from the calibration unit, and the simulation scenario.
 39. Thesimulation method of claim 28, further comprising pre-loading the datafrom the calibration unit into the calibration unit before the data fromthe calibration unit is received by the data processing unit.
 40. Thesimulation method of claim 28, further comprising: receiving, with adata processing unit, data from a tool calibration unit indicating aposition and/or orientation of a tool position and orientation sensorrelative to a tool; receiving, with the data processing unit, data fromthe tool position and orientation sensor indicating a position and/ororientation of the tool; generating, with the data processing unit, atool virtual image corresponding to the tool using the data from thetool position and orientation sensor and the data from the toolcalibration unit; and rendering, with the data processing unit, the toolvirtual image to a display.
 41. The simulation method of claim 40.further comprising: receiving, with a tracking unit in communicationwith the data processing unit, the data from the tool position andorientation sensor; and transmitting, with the tracking unit, the datafrom the tool position and orientation sensor to the data processingunit.
 42. The simulation method of claim 40, further comprisingselecting, with the data processing unit, a simulation scenario usingthe data from the tool position and orientation sensor and/or the datafrom the tool calibration unit.
 43. The simulation method of claim 42,further comprising: displaying, with the data processing unit, thesimulation scenario on the display; and receiving, with the dataprocessing unit, a selection of the simulation scenario.
 44. Thesimulation method of claim 42, further comprising generating, with thedata processing unit, a tool virtual image corresponding to the toolusing the data from the tool position and orientation sensor, the datafrom the tool calibration unit, and the simulation scenario.
 45. Thesimulation method of claim 40, further comprising pre-loading the datafrom the tool calibration unit into the tool calibration unit before thedata from the tool calibration unit is received by the data processingunit.